Imaging apparatus and method for controlling the same

ABSTRACT

An imaging apparatus includes an image capturing unit including an image sensor capable of photoelectrically converting a subject image and configured to generate moving image data based on an output signal of the image sensor, a storage unit configured to store foreign substance information including information relating to at least a position and a size of a foreign substance adhered to an optical element disposed on a front side of the image sensor, a detection unit configured to detect a shake amount of the image sensor, a control unit configured to control an image clipping position on an entire screen of the image sensor according to the shake amount of the image sensor detected by the detection unit, and a recording unit configured to record the foreign substance information and information indicating the image clipping position in association with the moving image data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/562,900 filed Sep. 18, 2009, which claims priority to Japanese PatentApplication No. 2008-244942 filed Sep. 24, 2008. Each of U.S. patentapplication Ser. No. 12/562,900 and Japanese Patent Application No.2008-244942 is hereby incorporated by reference herein in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an imaging apparatus using an imagesensor such as a charge coupled device (CCD) sensor or a complementarymetal oxide semiconductor (CMOS) sensor, and more specifically relatesto a technique capable of suppressing image quality from beingdeteriorated by a foreign substance adhered to a surface of an opticallow-pass filter disposed on the front side of the image sensor. Inparticular, the present invention relates to a technique capable ofsuppressing the image quality from being deteriorated by a foreignsubstance in a moving image capturing operation.

Further, the present invention relates to an imaging apparatus includingan electronic image stabilization system that performs shake correctionby electrically clipping a part of a captured image based on camerashake information.

2. Description of the Related Art

In a state where an interchangeable lens is removed from a camera bodyof a digital camera, a floating dust may enter an inside space of thecamera body. Further, a shutter mechanism or other mechanically operablecomponents are disposed in the camera body. Therefore, a slug may begenerated in the camera body when these mechanism components operate.

If such a foreign substance (i.e., dust or slug) adheres to a surface ofthe optical low-pass filter disposed on the front side of the imagesensor, which is an optical element configuring an imaging unit of thedigital camera, a captured image may include a shade caused by theforeign substance. Namely, the quality of the captured image may bedeteriorated.

When a camera is a type using a silver-halide film, the camerasuccessively feeds the film by a predetermined amount every time when animage is captured. Therefore, it is quite rare that continuouslycaptured images include the shade of the same foreign substance at thesame position. On the other hand, the digital cameras are not configuredto feed a film every time when an image is captured. Therefore,continuously captured images may include the shade of the same foreignsubstance at the same position.

To solve the above-described problem, there may be a method forcorrecting a defective pixel corresponding to a position of a foreignsubstance based on signals of surrounding pixels. For example, as atechnique for correcting such a defective pixel, an image defectcorrection method discussed in Japanese Patent Application Laid-Open No.6-105241 proposes a method to correct pixel defects of an image sensor.

Furthermore, as discussed in Japanese Patent Application Laid-Open No.2004-242158, to simplify setting of pixel defect position information,an extension of an image file captured in a dust acquisition mode can bedifferentiated from that of a normally captured image. In this case, apersonal computer (i.e., PC) automatically identifies a dust informationimage based on the extension and corrects an image to be correctedaccording to the obtained information.

Various techniques are conventionally available to correct a camerashake of a digital video camera. In general, the camera shake correctionincludes detecting camera shake information (e.g., shake amount or shakedirection) of an imaging device (e.g., digital video camera) with anexternal sensor or through image processing, and moving a part of anoptical system to cancel the camera shake, or partly clipping the image,based on the detection result.

As discussed in Japanese Patent Application Laid-Open No. 06-98246, asan external sensor, an angular speed sensor represented by a vibrationgyroscope can be used to directly measure a camera shake amount of theimaging device. Further, as discussed in Japanese Patent ApplicationLaid-Open No. 05-7327, image processing can be used to detect a motionvector of a screen based on a plurality of captured images and detect acamera-shake amount.

As a method for correcting a camera shake, a part of an imaging lenssystem is moved in a direction perpendicular to an optical axis to shiftan image formed on an image sensor. As discussed in Japanese PatentApplication Laid-Open No. 6-105241, a variable angle prism is disposedon the front side of the imaging lens system and an apical angle of thevariable angle prism is moved to shift the image formed on the imagesensor.

The above-described optical camera shake correction techniques aredisadvantageous in cost because they require mechanical members, such asactuators and optical elements, although a dynamic range for thecorrection can be enlarged.

On the other hand, as discussed in Japanese Patent Application Laid-OpenNo. 05-7327, an image sensor having a larger size compared to anactually required image size is prepared for a moving image capturingoperation and the camera shake can be corrected by clipping a part of animage obtained by the image sensor according to a detected camera shakeamount (hereinafter, referred to as “electronic image stabilization”).The above-described camera shake correction based on image clipping doesnot require any mechanical members, and brings an effect of reducing thecost and is therefore widely used.

In the above-described situation, not only compact digital cameras butalso lens-interchangeable digital cameras are recently required torecord highly accurate moving images having higher resolutions.

However, as described above, the lens-interchangeable digital camera issubjected to an influence of dusts or slugs adhered to the surface ofits image sensor due to various factors. Therefore, if a moving imagerecording is performed with the lens-interchangeable digital camera, aforeign substance may be constantly displayed during a playback (i.e.,reproduction) operation of the captured moving image.

According to a conventional dust removal method for thelens-interchangeable digital camera, dust removal relevant information(e.g., dust position and size information) and image data are recordedbeforehand. Subsequently, a personal computer performs image processingon captured images to remove the influence of the dust. In this case,the recorded image data contain dust components.

An electronic image stabilization system can be used to correct an imageshake generated by the camera shake when a moving image capturingoperation is performed. However, when the electronic image stabilizationsystem performs correction, a relative relationship between the dustremoval relevant information and an actual position of a dust on acaptured image may change because an image clipping position is variabledepending on each frame. Therefore, the electronic image stabilizationsystem cannot accurately perform dust correction processing.

SUMMARY OF THE INVENTION

According to an aspect of the present invention, an imaging apparatusincludes an image capturing unit having an image sensor capable ofphotoelectrically converting a subject image and configured to generatemoving image data based on an output signal of the image sensor; anacquisition unit configured to acquire foreign substance informationincluding information relating to at least a position and a size of aforeign substance adhered to an optical element disposed on a front sideof the image sensor; a detection unit configured to detect a shakeamount of the image sensor; a control unit configured to control animage clipping position on an entire screen of the image sensoraccording to the shake amount of the image sensor detected by thedetection unit; and a recording unit configured to record informationindicating the image clipping position in association with the movingimage data.

Further features and aspects of the present invention will becomeapparent from the following detailed description of exemplaryembodiments with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate exemplary embodiments, features,and aspects of the invention and, together with the description, serveto explain the principles of the invention.

FIG. 1 is a block diagram illustrating an example of a configuration ofan imaging apparatus having an image processing function according to afirst exemplary embodiment of the present invention.

FIG. 2 is a flowchart illustrating an example of processing performed bythe imaging apparatus (e.g., a digital camera according to the presentexemplary embodiment) to acquire dust information.

FIG. 3 illustrates a list of setting parameters to be used inacquisition of the dust information.

FIG. 4 illustrates an example of dust area size calculation.

FIG. 5 illustrates an example of a structure of a dust informationprofile.

FIG. 6 is a flowchart illustrating an example of still image capturingprocessing in an ordinary shooting operation according to an exemplaryembodiment of the present invention.

FIG. 7 is a flowchart illustrating an example of dust removalprocessing.

FIG. 8 is a flowchart illustrating an example of an interpolationroutine.

FIGS. 9A, 9B, and 9C illustrate an example of processing performed by ashake correction unit.

FIG. 10 illustrates an example of storage image clipping processing thatcan be performed by the shake correction unit.

FIG. 11 illustrates examples of meta data and media data according to anMP4 or similar file format.

FIG. 12 illustrates an example of a fragmented movie.

FIG. 13 is a flowchart illustrating an example of moving image capturingprocessing.

FIG. 14 is a flowchart illustrating an example of a recording routineaccording to the first exemplary embodiment.

FIG. 15 is a flowchart illustrating an example of a dust removalprocessing routine.

FIG. 16 illustrates an example of image clipping position recording onthe entire screen of an image sensor.

FIG. 17 illustrates an example of image clipping position recordingbased on a difference amount relative to a reference frame.

FIG. 18 illustrates an example of dust position coordinate conversion.

FIG. 19 is a flowchart illustrating an example of a recording routineaccording to a second exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments, features, and aspects of the presentinvention will now be herein described in detail below with reference tothe drawings. The relative arrangement of the components, the numericalexpressions, and numerical values set forth in these embodiments are notintended to limit the scope of the present invention.

FIG. 1 is a block diagram illustrating an example configuration of animaging apparatus having an image processing function according to afirst exemplary embodiment of the present invention.

In the present exemplary embodiment, the imaging apparatus is asingle-lens reflex type digital still camera that uses aninterchangeable lens. The present exemplary embodiment can be alsoapplied to a digital video camera using an interchangeable lens that canserve as an imaging apparatus.

As illustrated in FIG. 1, the imaging apparatus according to the presentexemplary embodiment includes a camera body 100 and aninterchangeable-lens type lens unit 300.

The lens unit 300 includes an imaging lens 310 including two or morelenses, a diaphragm 312, and a lens mount 306 that mechanically connectsthe lens unit 300 to the camera body 100. The lens mount 306 has variousfunctions for electrically connecting the lens unit 300 to the camerabody 100.

An interface 320, which is disposed on the lens mount 306, allows thelens unit 300 and the camera body 100 to perform data communications. Aconnector 322 can electrically connect the lens unit 300 to the camerabody 100.

The connector 322 allows the camera body 100 and the lens unit 300 totransmit control signals, state signals, and data signals. The connector322 can further function as a device for supplying electric power fromthe camera body 100 to the lens unit 300. The connector 322 is notlimited to electric communications and may be configured to performoptical communications and audio communications.

A diaphragm control unit 340 can control the diaphragm 312 inassociation with a shutter control unit 40 that controls a shutter 12 ofthe camera body 100, based on light metering information from a lightmetering control unit 46. A focus control unit 342 can control focusingof the imaging lens 310. A zoom control unit 344 can control zooming ofthe imaging lens 310.

A lens system control circuit 350 can control various operations thatare performed by the lens unit 300. The lens system control circuit 350includes a memory that stores operation constants, variables, andprograms. The lens system control circuit 350 further includes anonvolatile memory that stores identification information (e.g., anumber unique to the lens unit 300), administration information,function information (e.g., opened aperture value, minimum aperturevalue, and focal length), as well as present and past setting values.

An example of a configuration of the camera body 100 is described below.

A lens mount 106 can mechanically connect the camera body 100 to thelens unit 300. A pair of mirrors 130 and 132, which is a single-lensreflex type, can guide light incident on the imaging lens 310 to anoptical finder 104. The mirror 130 may be a quick return mirror or ahalf mirror.

The shutter 12 is a focal plane type shutter. The image sensor 14 is,for example, a CCD or CMOS sensor, which can photoelectrically convert asubject image. An optical element 14 a, such as an optical low-passfilter, is disposed on the front side of the image sensor 14. If aforeign substance, such as a dust, adheres to a surface of the opticalelement 14 a, an image captured by the image sensor 14 may include theforeign substance. In other words, the foreign substance on the opticalelement 14 a deteriorates image quality. The present exemplaryembodiment is directed to a technique capable of suppressing the imagequality from being deteriorated.

The light incident on the imaging lens 310 is guided via the diaphragm312 (i.e., a light quantity limiting element), the lens mounts 306 and106, the mirror 130, and the shutter 12, which serve as a single-lensreflex type optical system, to the image sensor 14 on which an opticalimage is formed.

An analog-digital (A/D) converter 16 can convert an analog signal outputfrom the image sensor 14 into a digital signal. A timing generationcircuit 18 can supply clock signals and control signals to the imagesensor 14, the A/D converter 16, and a digital-analog (D/A) converter26. A memory control circuit 22 and a system control circuit 50 cancontrol the timing generation circuit 18.

An image processing circuit 20 performs predetermined pixelinterpolation processing and color conversion processing on the dataoutput from the A/D converter 16 or data from the memory control circuit22. The image processing circuit 20 further performs predeterminedcalculation processing on image data output from the A/D converter 16,if useful.

The system control circuit 50 can perform through-the-lens (TTL) typeautomatic focus (AF) processing, automatic exposure (AE) processing, andflash pre-flash (EF) processing based on obtained calculation results,to control the shutter control unit 40 and the focus adjustment unit 42.

The image processing circuit 20 further performs predeterminedcalculation processing using the image data output from the A/Dconverter 16 and performs TTL automatic white balance (i.e., AWB)processing based on obtained calculation results.

The imaging apparatus according to the present exemplary embodimentillustrated in FIG. 1 includes the focus adjustment unit 42 and thelight metering control unit 46, which perform the AF processing, the AEprocessing, and the EF processing. In this case, the image processingcircuit 20 is not used for the AF processing, the AE processing, and theEF processing.

Alternatively, when the focus adjustment unit 42 and the light meteringcontrol unit 46 are used to perform the AF processing, the AEprocessing, and the EF processing, the image processing circuit 20 mayalso be operated to perform the AF processing, the AE processing, andthe EF processing.

The memory control circuit 22 can control the A/D converter 16, thetiming generation circuit 18, the image processing circuit 20, an imagedisplay memory 24, the D/A converter 26, a memory 30, and acompression/expansion circuit 32.

The image data output from the A/D converter 16 is written into theimage display memory 24 or the memory 30 via the image processingcircuit 20 and the memory control circuit 22 or via the memory controlcircuit 22 only.

An image display unit 28 includes a TFT liquid crystal display (LCD).Image data to be displayed, which is written in the image display memory24, is displayed on the image display unit 28 via the D/A converter 26.An electronic viewfinder (EVF) function can be realized by successivelydisplaying captured image data on the image display unit 28.

The image display unit 28 can turn on and off its display in response toan instruction from the system control circuit 50. When the display ofthe image display unit 28 is turned off, electric power consumption inthe camera body 100 can be greatly reduced.

The memory 30 can store captured still images and moving images. Thememory 30 has a storage capacity sufficient for storing a predeterminednumber of still images or predetermined amount of moving images.Therefore, even in a continuous shooting operation or in a panoramashooting operation, in which a plurality of still images arecontinuously captured, the digital camera can speedily write a greatamount of images into the memory 30.

Further, in a moving image capturing operation, the memory 30 can beused as a frame buffer for images continuously written at apredetermined rate. Furthermore, the memory 30 can be used as a workarea for the system control circuit 50.

A dust removal circuit 31 can perform image processing for removing dustfrom the image data referring to dust information stored in anonvolatile memory 56 and optical information obtained from the lensunit 300.

A compression/expansion circuit 32 can compress and expand the imagedata according to a conventional method. More specifically, thecompression/expansion circuit 32 reads an image from the memory 30,performs compression or expansion processing on the read image, andwrites the processed data into the memory 30.

Further, the compression/expansion circuit 32 has a function ofcompressing and encoding moving image data into a predetermined format,or expanding a moving image signal from predetermined encodedcompression data.

An audio signal processing circuit 33 can encode an audio signal inputfrom a microphone (not illustrated) into a predetermined encoded format,or can decode an audio signal from a predetermined encoded data.

The digital camera according to the present exemplary embodiment has afunction of outputting audio data decoded by the audio signal processingcircuit 33 via a speaker (not illustrated).

The shutter control unit 40 can control the shutter 12 in associationwith the diaphragm control unit 340 that controls the diaphragm 312based on the light metering information from the light metering controlunit 46.

The focus adjustment unit 42 can perform the automatic focus (AF)processing. The focus adjustment unit 42 measures a focus state of anoptical image that is formed when the light incident on the imaging lens310 in the lens unit 300 travels via the diaphragm 312, the lens mounts306 and 106, the mirror 130, and a focus adjustment sub mirror (notillustrated) according to the single-lens reflex mechanism.

The light metering control unit 46 can perform the automatic exposure(AE) processing. The light metering control unit 46 measures an exposurestate of an optical image that is formed when the light incident on theimaging lens 310 in the lens unit 300 travels via the diaphragm 312, thelens mounts 306 and 106, the mirror 130, and a light metering sub mirror(not illustrated) according to the single-lens reflex mechanism.

A flash 48 has an AF auxiliary light emission function and a flash lightadjustment function. The light metering control unit 46 and the flash 48can cooperatively perform flash light adjustment (EF) processing.

The digital camera can perform AF control based on measurement resultsobtained by the focus adjustment unit 42 and calculation resultsobtained by the image processing circuit 20 that processes the imagedata output from the A/D converter 16.

The digital camera can further perform exposure control based onmeasurement results obtained by the light metering control unit 46 andthe calculation results obtained by the image processing circuit 20 thatprocesses the image data output from the A/D converter 16.

The system control circuit 50, including a conventionally known centralprocessing unit (CPU), can control various operations to be performed bythe camera body 100. A memory 52 stores operation constants, variables,and programs to be used in the system control circuit 50.

A shake detection unit 53 can detect a camera shake of the camera body100. For example, the shake detection unit 53 includes an angular speedsensor (e.g., a vibration gyroscope).

A notification unit 54 can notify an operation state and a message,using text, image, and audio, according to execution of a program in thesystem control circuit 50. The notification unit 54 is, for example, adisplay device (e.g., LCD or light-emitting diode (LED)) configured toperform a visual display or an audio device that can generate sounds.The notification unit 54 can be configured as at least one of them or acombination of the above-described devices.

The notification unit 54 may include one or more easily visible displaydevices disposed in the vicinity of an operation unit 70 of the camerabody 100. The notification unit 54 may be partly incorporated in theoptical finder 104.

The notification unit 54 can display the following contents on the imagedisplay unit 28 (e.g., LCD).

The contents displayed on the image display unit 28 via the notificationunit 54 include shooting-mode related display, such as singleshooting/continuous shooting operation display, self timer display,etc., recording related display, such as compression rate display,recording pixel number display, number of recorded images display,number of recordable images display, etc., and shootingcondition-related display, such as shutter speed display, aperture valuedisplay, exposure compensation display, light adjustment correctiondisplay, external flash light emission amount display, red-eye reductiondisplay, etc.

Further, the contents displayed on the image display unit 28 includemiscellaneous display, such as macro shooting display, buzzer settingdisplay, battery remaining capacity display, error display, plural digitinformation display, attachment/detachment state display for a recordingmedium 200 and a personal computer (PC) 210, attachment/detachment statedisplay for the lens unit 300, communication I/F operation display,date/time display, connection state display for an external computer.

The display contents displayed on the optical finder 104 via thenotification unit 54 include, for example, in-focus display, shootingpreparation completion display, camera shake warning display, flash unitcharge display, flash charge completion display, shutter speed display,aperture value display, exposure compensation display, and recordingmedium writing operation display.

A nonvolatile memory 56 is an electrically erasable/recording memory(e.g., Electrically Erasable Programmable Read Only Memory (EEPROM))that stores programs. An optical information storage memory 58 storesvarious lens information obtained from the lens unit 300 via theconnector 122.

A mode dial switch 60, a shutter switch SW1 62, a shutter switch SW2 64,a playback switch 66, a single/continuous shooting switch 68, and theoperation unit 70 enables users to input various operation instructionsto the system control circuit 50. The operation input units for thesystem control circuit 50 can be configured as switches, dials, a touchpanel, a line-of-sight detection pointing device, and an audiorecognition device, which may be used as an independent component or acombination of a plurality of components.

Respective operation input units are described below in more detail.

The mode dial switch 60 can be operated to set a function shooting mode,such as an automatic shooting mode, a programmed shooting mode, ashutter speed priority shooting mode, an aperture priority shootingmode, a manual shooting mode, or a depth-of-focus priority shootingmode.

The mode dial switch 60 can be further used to set another functionshooting mode, such as a portrait shooting mode, a landscape shootingmode, a close-up shooting mode, a sports shooting mode, a night viewshooting mode, or a panoramic shooting mode.

If a shutter button (not illustrated) is pressed by a predeterminedamount (e.g., half-pressed), the shutter switch SW1 62 is turned on. Theshutter switch SW1 can be operated to instruct starting an operationrelating to the AF processing, the AE processing, the AWB processing, orthe EF processing.

If the shutter button (not illustrated) is fully pressed by a fullstroke, the shutter switch SW2 64 is turned on. The shutter switch SW264 can be operated to instruct starting sequential processing includingexposure processing, development processing, and recording processing.

First, in the exposure processing, the digital camera writes a signalread by the image sensor 14 into the memory 30 via the A/D converter 16and the memory control circuit 22. Then, the digital camera performsdevelopment processing using calculation results obtained by the imageprocessing circuit 20 and the memory control circuit 22.

Further, in the recording processing, the compression/expansion circuit32 performs compression on image data read from the memory 30 and thecompressed data is written in the recording medium 200 or transmitted tothe PC 210.

The playback switch 66 can be operated to instruct starting a playbackoperation for reading images from the memory 30, the recording medium200, or the PC 210, which stores the images captured in a shooting modestate, and displaying the read images on the image display unit 28. Theplayback switch 66 can be further used to set a function mode, such as aplayback mode, a multi-screen reproduction/deletion mode, or a PCconnection mode.

The single/continuous shooting switch 68 can be operated to select asingle shooting mode for capturing a one-frame image when the shutterswitch SW2 (64) is pressed and keeping a standby state for the nextshooting operation, or a continuous shooting mode for continuouslyperforming image capturing operations when the shutter switch SW2 (64)is pressed and held.

The operation unit 70 includes various buttons and a touch panel. Forexample, the operation unit 70 includes a live view start/stop button, amoving image recording start/stop button, a menu button, a set button, amulti-screen reproduction page-break button, a flash setting button, asingle shooting/continuous shooting/self timer switching button, a menushift + (plus) button, and a menu shift − (minus) button.

The operation unit 70 further includes a playback image shift + (plus)button, a playback image shift − (minus) button, a captured imagequality selection button, an exposure compensation button, a lightadjustment correction button, an external flash light emission amountsetting button, and a date/time setting button.

If a rotary dial switch is provided for each function of theabove-described plus button and the minus button, numerical values andfunctions can be easily selected.

The operation unit 70 further includes an image display ON/OFF switchthat can be operated to set ON/OFF of the image display unit 28, and aquick view ON/OFF switch that can be operated to set a quick viewfunction for automatically reproducing captured image data immediatelyafter the capturing operation is completed. The operation unit 70further includes a compression mode switch that can be operated to set acompression rate in a JPEG compression or select a RAW mode for directlyrecording a signal of the image sensor as a digital signal into arecording medium.

The operation unit 70 further includes an AF mode setting switch thatcan be operated to set a one-shot AF mode or a servo AF mode. In theone-shot AF mode, the digital camera starts an automatic focusingoperation in response to a turned-on of the shutter switch SW1 62, andif it attains an in-focus state, maintains the in-focus state. In theservo AF mode, the digital camera continuously performs the automaticfocusing operation when the shutter switch SW1 62 is pressed.

Furthermore, the operation unit 70 includes a setting switch that can beoperated to set a dust information acquisition mode for acquiring dustinformation from a captured dust detection image.

A power source switch 72 can be operated to switch a mode settingbetween on and off for the power source of the camera body 100. Thepower source switch 72 can be further operated to switch a mode settingbetween on and off for power sources of various devices (e.g., the lensunit 300, an external flash device 112, the recording medium 200, andthe PC 210) connected to the camera body 100.

A power source control unit 80 includes a battery detection circuit, adirect-current to direct-current (DC-DC) converter, a power supply blockswitch circuit. The power source control unit 80 detects a batteryattached to the camera body, a type of the battery, and a remainingcapacity of the battery. The power source control unit 80 controls theDC-DC converter based on detection results and an instruction from thesystem control circuit 50 so that electric power of a required voltageand a required time period can be supplied to various portions includinga recording medium.

A power source unit 86, which includes a primary battery (e.g., alkalibattery or a lithium battery), a secondary battery (e.g., a NiCdbattery, a NiMH battery, a Li-ion battery, or a Li polymer battery), andan AC adapter, is connected to the camera body 100 via a pair ofconnectors 82 and 84.

An interface 90 is connectable via a connector 92 to the recordingmedium 200 (e.g., a memory card or a hard disk). An interface 94 isconnectable via a connector 96 to the PC 210. A recording mediumattachment/detachment detection circuit 98 can detect whether therecording medium 200 or the PC 210 is attached to the connector 92and/or 96.

In the present exemplary embodiment, two sets of the interface and theconnector are provided for attaching recording media. However, the totalnumber of the interfaces and the connectors can be appropriatelychanged. The interfaces and the connectors can be arbitrarily combinedwith any other type of interfaces and connectors.

The interfaces and the connectors can be configured to conform tostandards of various storage media. For example, a Personal ComputerMemory Card International Association (PCMCIA) card or a compact flash(CF®) card, or a SD card can be used. If the interfaces 90 and 94 andthe connectors 92 and 96 are configured to conform to the standards ofthe PCMCIA card or the CF card, various communication cards can beconnected to the digital camera.

The communication cards connectable to the camera body 100 include alocal area network (LAN) card, a modem card, a Universal Serial Bus(USB) card, an Institute of Electrical and Electronic Engineers (IEEE)1394 card, a P1284 card, a Small Computer System Interface (SCSI) card,and a PHS. The digital camera can transfer image data and relevantadministration information, via any one of the above-describedcommunication cards, to peripheral devices including other computers andprinters.

The optical finder 104 can receive light incident on the imaging lens310 guided by the single-lens reflex mechanism (i.e., the diaphragm 312,the lens mounts 306 and 106, and the mirrors 130 and 132). An opticalimage can be displayed on the optical finder 104. Thus, the digitalcamera enables users to perform a shooting operation using only theoptical finder without using an electronic finder function realized bythe image display unit 28.

At least part of the functions of the notification unit 54, such asdisplays relating to in-focus status, camera shake warning, flashcharging, shutter speed, aperture value, and exposure compensation, canbe realized by the optical finder 104.

The external flash device 112 is attached to the camera body 100 via anaccessory shoe 110.

An interface 120 connects the camera body 100 to the lens unit 300 inthe lens mount 106.

The connector 122 electrically connects the camera body 100 to the lensunit 300. A lens attachment/detachment detection unit (not illustrated)can detect whether the lens unit 300 is mounted on the lens mount 106and the connector 122.

Control signals, state signals, and data signals can be transmitted viathe connector 122 between the camera body 100 and the lens unit 300.Electric power can be also supplied from the camera body 100 to the lensunit 300 via the connector 122.

Optical information (e.g., aperture, zoom position, pupil position, andfocal length) of the lens unit 300 transmitted via the connector 122 canbe stored in the optical information storage memory 58 of the camerabody 100. Each of the camera body side and the lens unit side canrequest starting communication in response to a renewal of theinformation.

The connector 122 can be configured to perform optical communicationsand audio communications.

The recording medium 200 is, for example, a memory card or a hard disk.The recording medium 200 includes a recording unit 202 configured by asemiconductor memory or a magnetic disk, an interface 204 forcommunication with the camera body 100, and a connector 206 forconnection with the camera body 100.

The recording medium 200 may be configured by the PCMCIA card, thecompact flash®, a micro digital audio tape (DAT), a magneto-opticaldisk, a compact disc recordable (CD-R), a compact disc-rewritable(CD-RW), or other comparable optical disk, a digital versatile disc(DVD), or other comparable phase change optical disc.

The PC 210 includes a recording unit 212 configured by a magnetic disk(HD), an interface 214 for communication with the camera body 100, and aconnector 216 for connection with the camera body 100. The interface 94may be a USB type or an IEEE1394 type, although not limited to aspecific type.

The imaging apparatus having the above-described configuration canperform the following image processing for removing an influence of adust adhered to an optical element (e.g., a low-pass filter or a coverglass) disposed on the front side of the image sensor.

In the present exemplary embodiment, the imaging apparatus captures adust detection image to be used to obtain dust information (i.e.,foreign substance information), which indicates a position and the sizeof a dust (i.e., a foreign substance). Then, the imaging apparatusextracts and generates dust data.

It is desired that the dust detection image is an image of an objecthaving a uniform luminance surface. However, the uniformity is not sostrictly required because it is desired that the dust detection imagecan be easily obtained. For example, the dust detection image is a bluesky or a whitewall surface.

FIG. 2 is a flowchart illustrating an example of processing that isperformed by the imaging apparatus (e.g., the digital camera accordingto the present exemplary embodiment) to acquire the dust information inthe present exemplary embodiment.

First, in step S201, the imaging apparatus determines whether the dustinformation acquisition mode is selected by the operation unit 70. Ifthe determination in step S201 is NO, the imaging apparatus repeats theprocessing in step S201 until the dust information acquisition mode isselected. If it is determined that the dust information acquisition modeis selected (YES in step S201), then in step S202, the imaging apparatusdetermines whether the shutter switch SW1 62 is turned on. If it isdetermined that the shutter switch SW1 62 is in an OFF state (NO in stepS202), the processing returns to step S201 to repeat the above-describedprocessing.

On the other hand, if it is determined that the shutter switch SW1 62 isin an ON state (YES in step S202), then in step S203, the imagingapparatus sets an aperture value, an ISO value, a shutter speed, andother shooting-related parameters. FIG. 3 illustrates examples of thesetting parameters.

The aperture value is set to F22 according to which an aperture of thediaphragm is decreased. For example, it is useful to set a stopped-downaperture value in a range settable by the lens unit 300 connected to thelens mount 106.

In general, the dust adheres to a protection glass of the image sensor14 (not a surface of the image sensor 14 itself) or an optical filter(i.e., the optical element 14 a) disposed on the subject side of theimage sensor. Accordingly, an image formation state is variabledepending on the aperture value of the lens unit 300.

If the diaphragm is in or close to an opened state, an image of the dustmay be defocused. An appropriate dust detection image cannot beacquired. It is therefore desired to set a stopped-down aperture value.

Referring back to the flowchart illustrated in FIG. 2, a photographerdirects the imaging apparatus toward a white wall or another uniformluminance surface and presses the shutter switch SW2 64.

In step S204, the imaging apparatus determines whether the shutterswitch SW2 64 is turned on. If it is determined that the shutter switchSW2 64 is in an OFF state (NO in step S204), the processing returns tostep S202 to perform the above-described determination related to theshutter switch SW1 62. If it is determined that the shutter switch SW264 is in an ON state (YES in step S204), the processing proceeds to stepS205.

In step S205, the imaging apparatus captures the dust detection image(e.g., the uniform luminance surface) and stores captured image data inthe memory 30. Next, in step S206, the imaging apparatus acquires thedust information from the image data stored in the memory 30.

The imaging apparatus performs the following processing for acquiringthe dust information. More specifically, the imaging apparatus obtainsposition (i.e., coordinates) and size information of a dust area fromthe captured dust detection image.

First, the imaging apparatus divides the area of the captured dustdetection image into a plurality of blocks, calculates a maximumluminance Lmax and an average luminance Lave in each block, andcalculates a threshold value T1 in the block according to the followingformula.

T1=Lave×0.6+Lmax×0.4

Next, the imaging apparatus specifies a pixel not exceeding thethreshold value T1 as a dust pixel because a pixel, to which a dust isattached, has a luminance value lower than those of surrounding pixels.Then, the imaging apparatus identifies an independent area configured bythe dust pixel(s) as a dust area di (i=0, 1, . . . ,n).

FIG. 4 illustrates an example of dust area size calculation. Asillustrated in FIG. 4, the imaging apparatus obtains a maximum valueXmax and a minimum value Xmin in the horizontal direction as well as amaximum value Ymax and a minimum value Ymin in the vertical direction,with respect to the coordinate values of the pixel configuring each dustarea, and calculates a radius ri representing the size of the dust areadi according to the following formula.

ri=[√{(Xmax−Xmin)²+(Ymax−Ymin)²}]/2

In this case, center coordinate values (Xdi, Ydi) can be approximatelydefined by the following formulae.

Xdi=(Xmax+Xmin)/2

Ydi=(Ymax+Ymin)/2

The imaging apparatus records the obtained position (i.e., coordinatevalues) and the radius as a dust information profile.

Due to the restriction in the size of the nonvolatile memory 56, dustcorrection data (i.e., dust information profile) may be limited in itsdata size. In such a case, the imaging apparatus sorts dust positioninformation according to the size or an average luminance value of thedust area.

In the present exemplary embodiment, the imaging apparatus sorts thedust position information in descending order of the magnitude of theradius ri. If two or more pieces of the dust position information cannotbe discriminated in the magnitude of the radius ri, then the imagingapparatus sorts them according to ascending order of the magnitude ofthe average luminance value. Through the above-described processing, theimaging apparatus can prioritize an outstanding dust over others inregistration of the dust correction data. In the present exemplaryembodiment, Di represents a sorted dust area and Ri represents a radiusof the dust area Di.

If a target dust area has a size greater than a predetermined size, thetarget dust area can be removed from the candidates to be sorted and canbe relocated on the tail of a sorted dust area list. If a large dustarea is subjected to interpolation processing, the image quality maydeteriorate. Therefore, it is desired to rank the large dust area lowestin the priority order of the correction target.

FIG. 5 illustrates an example of a structure of the dust informationprofile. As illustrated in FIG. 5, the dust information profile storeslens information and dust position/size information in a dust detectionimage capturing operation. More specifically, as the lens information inthe dust detection image capturing operation, the dust informationprofile stores an actual aperture value (i.e., F-number) and acorresponding lens pupil position in the dust detection image capturingoperation.

Subsequently, the imaging apparatus stores a number of detected dustareas (i.e., an integer value) in a storage area and then repetitivelystores parameters representing respective dust areas. The parametersrepresenting each dust area can be a set of three numerical values,e.g., radius of dust (e.g., 2 bytes), X coordinate (e.g., 2 bytes) and Ycoordinate (e.g., 2 bytes) of the center of an effective image area.

In step S207, the imaging apparatus stores the acquired dust informationin the nonvolatile memory 56. Then, the imaging apparatus terminates thedust information acquisition processing.

A shooting operation in the dust information acquisition mode isintended to acquire the dust information. Therefore, in the presentexemplary embodiment, the imaging apparatus does not compress a capturedimage and does not record the image into the recording medium 200.

The above-described processing is effective to prevent the capacity ofthe recording medium 200 from being uselessly used for the image dataunnecessary for the photographer. However, similar to an ordinary image,the image can be compressed and then stored in the recording medium 200.Further, in this case, an extension may be changed.

The present exemplary embodiment relates to a correction method to beperformed when a moving image is captured, including image processingthat can correct the image quality deteriorated by a dust. Prior to adetailed description of the moving image processing, an example of stillimage processing is described below.

In a case where an ordinary shooting operation (which is different fromthe dust detection image capturing operation) is performed, if the imageto be processed is a still image, the imaging apparatus records the dustcorrection data (i.e., the dust information profile) illustrated in FIG.5 in association with image data, in addition to camera setting valuesin the ordinary shooting operation, in the recording medium 200.

More specifically, to realize the associated recording, the imagingapparatus can additionally write the dust correction data, for example,in an Exchangeable image file format (Exif) area (i.e., a header area)of an image file in which the camera setting values in the shootingoperation are recorded.

Alternatively, to realize the associated recording, the imagingapparatus can record the dust correction data independent of the fileand record only link information of the dust correction data file in theimage data.

However, if the image file and a dust correction data file areseparately recorded, the link relationship may be lost when the imagefile is transferred. Therefore, it is desired to store the dustcorrection data integrated with the image data.

Recording the dust correction data in association with the image data isuseful when the image data including the additionally recorded dustcorrection data is transferred to an external image processingapparatus, to enable the external image processing apparatus to performthe dust removal processing.

Next, an example of the dust removal processing in the ordinary shootingoperation, using the dust information stored in the nonvolatile memory56 as described above, is described below with reference to flowchartsillustrated in FIGS. 6 and 7.

Although the following description relates to an example of the dustremoval processing for a still image, similar dust removal processingcan be executed for each frame of a moving image.

FIG. 6 is a flowchart illustrating an example of still image capturingprocessing in an ordinary shooting operation according to the presentexemplary embodiment.

In step S501, the imaging apparatus determines whether the shutterswitch SW1 62 is turned on. If it is determined that the shutter switchSW1 62 is in the OFF state (NO instep S501), the imaging apparatusrepeats the processing in step S501. If it is determined that theshutter switch SW1 62 is in an ON state (YES in step S501), then in stepS502, the imaging apparatus performs light metering and focus adjustmentprocessing.

Next, in step S503, the imaging apparatus determines whether the shutterswitch SW2 64 is turned on. If it is determined that the shutter switchSW2 64 is in the OFF state (NO in step S503), the processing returns tostep S501 to repeat the above-described processing. If it is determinedthat the shutter switch SW2 64 is in the ON state (YES in step S503),then in step S504, the imaging apparatus performs a shooting operation.

If the imaging apparatus completes the shooting operation, then in stepS505, the imaging apparatus determines whether any effective dustinformation is present in the nonvolatile memory 56. If it is determinedthat the dust information is present (YES in step S505), the processingproceeds to step S506. If it is determined that the dust information isabsent (NO in step S505), then in step S507, the imaging apparatusstores the captured image data in the recording medium 200.

In the present exemplary embodiment, the imaging apparatus determineswhether the dust information is present in the nonvolatile memory 56.However, the imaging apparatus can use any other method to determinewhether the shooting operation is performed in the above-described dustinformation acquisition mode.

For example, it is useful that the imaging apparatus evaluates a flagthat is set when the shooting operation is performed in the dustinformation acquisition mode.

In step S506, the imaging apparatus embeds the acquired dust informationin a header area (e.g., Exif area) of the captured image data. In stepS507, the imaging apparatus stores the image data including the embeddeddust information in the recording medium 200.

An example of the dust removal processing is described below withreference to FIG. 7.

In step S601, the imaging apparatus determines whether the selectedimage includes embedded dust information. If it is determined that theselected image includes the embedded dust information (YES in stepS601), then in step S602, the imaging apparatus acquire the dustinformation.

In step S603, the imaging apparatus performs correction processing toremove the influence of a dust on image data based on the captured dustinformation. For example, the imaging apparatus can perform pixelinterpolation processing applied to peripheral pixels around a dust.

More specifically, the imaging apparatus obtains a row of coordinatevalues Di (i=1, 2, . . . n), a row of radii Ri (i=1, 2, . . . , n), anaperture value f1, and a lens pupil position L1 from the extracted dustcorrection data. In the present embodiment, Ri represents the size of adust located on the coordinate Di, which can be preliminarily obtainedin a dust correction data sort operation. Further, f1 represents anaperture value of the lens in a dust detection image capturingoperation, and L1 represents a pupil position of the lens in the dustdetection image capturing operation.

The imaging apparatus further acquires an aperture value f2 and a lenspupil position L2 in an ordinary image capturing operation and convertsDi according to the following formula. In the present embodiment, drepresents the distance from the image center to the coordinate positionDi, and H represents the distance between a surface of the image sensor14 and the dust. For example, the following formulae can define aconverted coordinate Di′ and a converted radius Ri′.

Di′(x, y)=(L2×(L1−H)×d/((L2−H)×L1))×Di(x, y) Ri′=(Ri×f1/f2+3)  (1)

The unit in the above-described formulae is “pixel.” In the formula ofRi′, “+3” is a margin amount.

The imaging apparatus detects a dust in an area identified by thecoordinate Di′ and the radius Ri′ and, if useful, performs interpolationprocessing on pixels of the identified area, as described below. Theimaging apparatus performs the dust removal processing on all coordinatepositions and then the processing to step S604 if the dust removalprocessing is completed.

In step S604, the imaging apparatus newly records a correction processedimage (i.e., the captured image from which the influence of the dust hasbeen removed). Then, the imaging apparatus terminates the dust removalprocessing.

In the present exemplary embodiment, recording is performed in thecamera body 100 by embedding the dust information in the captured imagedata and, subsequently, dust influence removal correction processing isperformed.

Alternatively, the imaging apparatus can perform the dust influenceremoval correction processing without embedding the dust informationwhen the image is captured and recorded in the camera body 100, and thencan record the correction processed image in the recording medium 200.

An example of dust area interpolation processing is described below inmore detail. FIG. 8 is a flowchart illustrating detailed processing inan interpolation routine.

First, in step S701, the imaging apparatus performs a dust areadetermination. In the present exemplary embodiment, the dust area is anarea satisfying all of the following conditions:

(1) An area that is darker than a threshold value T2 defined by thefollowing formula, which is obtained by using an average luminance Yaveand a maximum luminance Ymax of pixels included in the center coordinateDi′ and the radius Ri′ (Di′ and Ri′ obtained according to the formula(1)).

T2=Yave×0.6+Ymax×0.4

(2) An area that does not contact a circle defined by the centercoordinate Di′ and the radius Ri′.(3) An area having a radius value not less than X1 and not greater thanX2, when the radius value is calculated according to the above-describedmethod for an isolated area constituted by low-luminance pixels selectedby (1).(4) An area that includes the center coordinate Di of the circle.

In the present exemplary embodiment, X1 is equal to three pixels and X2is equal to 30 pixels. Through the above-described screening, theimaging apparatus can process only a small isolated area as a dust area.If the lens pupil position cannot be accurately acquired, theabove-described condition (4) can be changed appropriately.

For example, if a target area includes coordinates in the range of ±3pixels from the coordinate Di in both the X direction and the Ydirection, the imaging apparatus determines it as a dust area.

In step S702, the imaging apparatus determines whether an image signalincludes the above-described dust area (portion). If it is determinedthat the image signal includes the dust area (YES instep S702), theninstep S703, the imaging apparatus performs dust area interpolationprocessing. If it is determined that the image signal does not includethe dust area (NO in step S702), the imaging apparatus terminates theprocessing of the interpolation routine.

The imaging apparatus can perform the dust area interpolation processingin step S703 according to a conventionally known defective areainterpolation method. For example, a pattern replacement discussed inJapanese Patent Application Laid-Open No. 2001-223894 can be used forthe defective area interpolation.

According to the method discussed in Japanese Patent ApplicationLaid-Open No. 2001-223894, a defective area is specified using aninfrared ray. In the present exemplary embodiment, the imaging apparatusspecifies the dust area detected in step S701 as a defective area andinterpolates the dust area using the pattern replacement method based onsurrounding normal pixels.

If any pixel cannot be interpolated using the pattern replacementmethod, the imaging apparatus can select a predetermined number ofnormal pixels from the pattern interpolated image data that are closestto a target pixel to be interpolated and can perform interpolation basedon an average color of the selected pixels.

In a case where the dust removal processing is applied to a still image,the above-described processing for attaching dust correction data toimage data is useful because it is unnecessary to identify arelationship between dust correction image data and captured image data.

The dust correction data is compact data constituted by position, size,and conversion data (aperture value, lens pupil position, etc.).Therefore, the data size of a captured image does not become so large.The possibility of error detection can be greatly reduced by performinginterpolation processing on only an area including pixels designated bythe dust correction data.

The imaging apparatus performs the following electronic imagestabilization processing for detecting a camera shake and clipping apart of an image obtained by the image sensor to correct the camerashake according to a camera shake amount.

First, a method for detecting a camera shake is described below. Forexample, a gyroscope (i.e., an angular speed sensor) attached to thecamera body 100 can physically detect a vibration of the camera and canoutput an angular speed signal representing the camera shake.

A high-pass filter for cutting direct-current (DC) removes a DCcomponent from the angular speed signal output from the angular speedsensor to allow only a vibration component to pass. Further, anintegrator integrates the vibration component output from the high-passfilter, calculates an average value of the vibration component, andoutputs an angular variation signal as an evaluation value thatrepresents the camera shake.

A shake correction unit is described below with reference to FIGS. 9A,9B, and 9C. In FIG. 9A, an area 260 indicates an entire imaging area ofthe image sensor 14. An area 261 surrounded by a dotted line is aclipping frame defined in the entire imaging area of the image sensor.An image in the clipping frame is converted into a standard video signalthat can be actually output. In this example, a main subject 269 iscaptured by a photographer.

FIG. 9C illustrates a video displayed using the standard video signal.As illustrated in FIG. 9C, a reproduced main subject 269′ is in adisplay area 265 of a monitor that reproduces the video signal.

To realize the display area 265 of the monitor, the imaging apparatusperforms processing for clipping a captured image to output the standardvideo signal representing the entire imaging area of the image sensor 14except its peripheral region.

FIG. 9B indicates a change of the image that occurs when thephotographer moves the camera in a direction indicated by arrows 262,262′, and 262″ (i.e., lower left direction). In this case, the subject269 shifts in a direction indicated by an arrow 264 (i.e ., upper rightdirection) on the entire imaging area 260 of the image sensor 14.

If the image is clipped using a clipping frame 261′ located on the sameposition as the clipping frame 261 illustrated in FIG. 9A, the imagingapparatus generates a video signal representing the subject shifted by avector amount corresponding to the arrow 264.

The imaging apparatus can obtain the video illustrated in FIG. 9C byshifting the clipping frame from the above-described frame position 261′to a frame position 261″ based on an image shift amount 263 obtainedfrom the shake amount of the imaging apparatus (i.e., a shake correctiontarget value). The present exemplary embodiment realizes an image shakecorrection based on the above-described principle.

Imaging area clipping processing that is performed by the imagingapparatus according to the present exemplary embodiment is describedbelow with reference to FIG. 10.

An entire area 360 of the image sensor 14 is constituted by a pluralityof pixels (i.e., photoelectric conversion elements) 361. The imagingapparatus controls charge accumulation and reading of each pixel basedon an electric driving pulse generated by a timing generator (notillustrated).

Two clipping frames 362 and 363 are similar to the clipping frame 261illustrated in FIGS. 9A, 9B, and 9C. For example, the imaging apparatuscan perform a video signal clipping using the clipping frame 362illustrated in FIG. 10 in the following manner.

First, the imaging apparatus successively reads a photoelectricallyconverted charge amount of each pixel from a starting pixel indicated by“S” in a direction indicated by an arrow 365. The imaging apparatusstarts the reading processing within a synchronization period of anoutput video signal, and terminates the reading processing at a pixelposition immediately before a pixel “A” before the synchronizationperiod ends. The transfer rate in the above-described reading processingis higher than that in the ordinary reading processing.

In an actual video period (i.e., after the synchronization period ends),the imaging apparatus reads a charge amount of each pixel from the pixel“A” to a pixel “F” at an ordinary reading speed as image information ofa one-line video signal.

Further, in a horizontal synchronization period before the next line,the imaging apparatus reads pixels from a pixel following the pixel “F”to a pixel positioned immediately before a pixel “G” at a transfer ratehigher than the ordinary reading speed. Then, the imaging apparatuswaits for reading of the next video period.

Similar to the above-described reading processing for the pixels from“A” to “F”, the imaging apparatus starts reading processing from thepixel “G.”

As described above, the imaging apparatus can control reading timing toselectively extract an arbitrary portion (e.g., a central portion) fromthe entire imaging area of the image sensor and can obtain a videosignal representing the extracted portion.

As illustrated in FIGS. 9A, 9B, and 9C, a captured image shifts in adirection corresponding to a movement of the imaging apparatus. Ashifting in the clipping position is described below with reference toFIG. 10.

If a shifting of the subject (=a shake of the imaging apparatus) on theimage sensor surface is detected and the shifting amount is comparableto an arrow 364, the imaging apparatus relocates the clipping frame fromthe position 362 to the position 363 to obtain a clipped video that isnot influenced by the shifting of the subject.

In this case, to change the clipping position, the imaging apparatuschanges the reading start position from “A” to “B” and reading finishposition from “c” to “D”, and the imaging apparatus selectively extracta partial portion of the image from the entire imaging area 360 of theimage sensor in the same manner as in the above-described readingprocessing starting from the pixel “A.”

More specifically, similar to the above-described reading processingusing the clipping frame 362, the imaging apparatus successively readsthe photoelectrically converted charge amount of each pixel from thestarting pixel indicated by “S” in the direction indicated by the arrow365.

The imaging apparatus starts the reading processing within thesynchronization period of the output video signal, and terminates thereading processing at a pixel position immediately before the pixel “B”before the synchronization period ends. The transfer rate in theabove-described reading processing is higher than that in the ordinaryreading processing. In the actual video period, the imaging apparatusstarts reading the charge amount of each pixel from the pixel “B” at anordinary transfer rate.

As described above, the imaging apparatus preliminarily reads aperipheral portion of the imaging area of the image sensor by an amountcorresponding to shake correction information during a synchronizationsignal period that does not appear in the actual video period. Theimaging apparatus selectively reads a part of the image sensor based onthe camera shake information and can obtain a video signal not includingan adverse influence of the camera shake.

The following is a moving image file format, which is generally referredto as “MP4” and can be used for recording of moving image data in adigital camera or a digital video camera.

The MP4 file format (see ISO/IEC 14496-14; “Informationtechnology—Coding of audio-visual objects—Part 14: MP4 file format”;ISO/IEC; 2003-11-24) is a file format expanded based on a general fileformat “ISO Base Media File Format” (see ISO/IEC 14496-12; “Informationtechnology—Coding of audio-visual objects—Part 12: ISO base media fileformat”; ISO/IEC; 2004-01-23), which is standardized by ISO/IECJTC1/SC29/WG11 (International Organization forStandardization/International Engineering Consortium) to record MPEG orother moving image/audio content data in a file.

The present exemplary embodiment is not limited to the MP4 file formatand can be also applied to a similar file format. For example, ISO fileformat standards, such as “Motion JPEG 2000 file format” (ISO/IEC15444-3) and “AVC file format” (ISO/IEC 14496-15), have a basicstructure similar to the MP4 file format.

FIG. 11 illustrates an example of a data structure according to the MP4file format.

An MP4 file 1001 includes a meta data (i.e., header information) 1002that indicates physical positions of video/audio data, temporalpositions, and characteristics information, and a media data 1003 thatindicates actual states of encoded video/audio data.

In the MP4 format, presentation of the entire content is referred to as“movie” and presentation of a media stream constituting the content isreferred to as “track.” The meta data 1002 includes a video track 1004that logically handles moving image data and an audio track 1005 thatlogically handles audio data. The video track 1004 and the audio track1005 are similar in configuration.

Namely, respective tracks can record various meta data information ofactual media data although their contents are variable depending on thecharacteristics of the media data.

For example, the video track 1004 stores configuration information of adecoder that decodes encoded data and information relating to arectangular size of a moving image.

In addition, the video track 1004 stores an offset 1006 indicating afile position where the media data is actually recorded, a sample size1007 indicating the size of each frame data (which may be referred to as“picture”) of the media data, and a time stamp 1008 indicating adecoding time of each frame data.

On the other hand, the media data 1003 records substances of the movingimage data and the audio data. A data structure of the media data 1003is generally referred to as a “chunk” that continuously records“samples” indicating a basic unit of encoded data.

The chunk includes, according to the track of the meta data 1002, avideo chunk 1009 including moving image media data and an audio chunk1010 including audio media data.

According to the configuration illustrated in FIG. 11, the video chunk1009 and the audio chunk 1010 are alternately recorded. However, therecording position and the order of these chunks may be arbitrarilychanged.

The illustrated example is an example of a generally recorded format.However, the above-described alternate layout (i.e., interleave) bringsan effect of improving accessibility to the data recorded in the filebecause a moving image and an audio data to be reproduced simultaneouslyare positioned closely. Therefore, the above-described alternate layoutis widely used.

The chunk includes one or more samples of individual media data. Forexample, as illustrated in FIG. 11, the video chunk 1009 includes aplurality of video samples (i.e., frames) 1011 continuously recorded.

In general, the video sample (i.e., frame) 1011 corresponds to a singleframe data (i.e., a picture) of a video. Respective tracks areassociated with chunks in the following manner.

For example, if the video track 1004 stores moving image data,information included in the video track 1004 includes informationrelating to each video chunk 1009 included in the media data 1003.

The offset 1006 is constituted by an information table indicating arelative position of respective video chunks 1009 on the file, so thatthe position of an actual video chunk can be known by referring to eachentry on the table.

The sample size 1007 is written as a size table of a plurality ofsamples included in respective chunks, i.e., the size of respectiveframes of a video. More specifically, the video track 1004 storesinformation indicating the number of samples included in each chunk, sothat the samples included in respective video chunks 1009 can beaccurately acquired based on the stored information.

The time stamp 1008 is constituted by a table recording a decoding timeof respective samples as a difference between samples, so that the timestamp of each sample can be acquired by calculating a cumulative time byreferring to this table.

The above-described track and chunk relationship is similarlyestablished between the audio track 1005 and the audio chunk 1010.Accordingly, in both the MP4 file format and the ISO Base Media FileFormat, encoded data can be acquired from an arbitrary position by arequired amount based on the meta data 1002 and the media data 1003 withadditional information such as the time stamp.

The present exemplary embodiment does not describe all of standardizedrecording information because detailed contents of standardizeddefinitions can be known by referring to the corresponding descriptionsin ISO/IEC 14496.

Data to be recorded in a file according to the MP4 file format isdescribed in a data structure, which is referred to as a “BOX.” The BOXis configured by the following fields:

Size: size of the entire BOX including a size field itself.

Type: 4-byte identifier representing a type of the BOX, which is usuallyexpressed using four alphanumeric characters.

Each BOX may include other fields as options, although not describedbelow.

Data to be recorded in a file is stored in a different type of BOXdepending on its type. For example, the media data 1003 is recorded asMedia Data BOX that stores encoded data (the content of a type field is‘mdat’ and, in the following description, if an identifier indicating aBOX type is used, the identifier expresses the BOX having the indicatedtype). The meta data 1002 is recorded as Movie BOX (′moov′) that storesmeta data information of the entire content.

Similarly, the above-described information relating to the chunk and thesample is recorded for each track in the moov as a BOX having a uniqueidentifier.

The MP4 file format allows not only recording all of meta data in themoov but also dividing the meta data into a plurality of areas andsequentially recording the divided data areas. The latter format isreferred to as “Fragmented Movie.”

FIG. 12 illustrates a file structure according to a fragmented movieformat. Contents of the fragmented movie format (such as media data andmeta data) can be divided into a plurality of “fragments” at arbitraryunits of time and time-sequentially recorded from the head of a file.For example, according to the example illustrated in FIG. 12, moov 1101indicates meta data in the first fragment and stores informationrelating to data included in mdat 1102.

Similarly, moof 1103 indicates meta data in the second fragment andstores information relating to data included in mdat 1104. In a casewhere the fragmented movie format is employed, a Movie Extends Box(‘mvex’) is added 1105 indicating that a fragment is present in the moov1101. Information included in the mvex 1105 is, for example, duration(time length) of the entire content including all fragments.

As described above, the file according to the MP4 file format storesvarious attributes relating to the media data in a meta data areaseparately from the media data. Therefore, it is easy to access desiredsample data regardless of a physical storage state of the media data.

An example of a method for associating the above-described dustcorrection data with the video sample (frame) 1011 in a moving imagerecording operation is described below, in which the moving image fileformat to be used for recording moving image and audio data according tothe present exemplary embodiment is the fragmented movie formatillustrated in FIG. 12 (i.e., the MP4 file format).

The present exemplary embodiment can be also applied to theabove-described standards “Motion JPEG 2000 file format” (ISO/IEC15444-3) and the “AVC file format” (ISO/IEC 14496-15), as well as anyother standards employing a file format and an architecture similar tothose regulated by MP4, such as a Third Generation Partnership Project(3GPP) file format, which is a moving image file standard regulated forwireless terminals represented by third generation portable phones (see3GPP TS 26.244 “Technical Specification Group Services and SystemAspects Transparent end-to-end packet switched streaming service (PSS);3GPP file format (3GP) (Release 6)” 3rd Generation Partnership Project;2003-02-28).

A file recording operation and dust removal processing that is performedin a moving image capturing operation by the imaging apparatus havingthe image processing function according to the present exemplaryembodiment is described below.

FIG. 13 is a flowchart illustrating an example of the moving imagecapturing processing.

First, in step S1401, the imaging apparatus determines whether live viewstart is selected via the operation unit 70. If it is determined thatthe live view start is not selected (NO in step S1401), the imagingapparatus repeats the determination processing in step S1401.

If it is determined that the live view start is selected (YES in stepS1401), the imaging apparatus drives the mirror 130 up to open theshutter 12 so that an optical image is formed on the image sensor 14.

The A/D converter 16 converts an analog signal output from the imagesensor 14 into a digital signal at a predetermined frame rate. The imageprocessing circuit 20 performs the predetermined pixel interpolationprocessing and the color conversion processing, and stores the processedsignal in a frame memory buffer of the memory 30.

In this case, the camera body 100 requests the lens unit 300 to transmitvarious optical information (e.g., aperture, zoom position, pupilposition, and focal length) according to the frame rate. The opticalinformation storage memory 58 stores various optical informationreceived from the lens unit 300 via the connector 122 in associationwith each image data of the frame memory buffer.

The image data stored in the frame memory buffer is read again by theimage processing circuit 20 that converts the read image data intodisplay image data (i.e., image data to be used for display). The imagedisplay memory 24 stores the display image data.

The image display unit 28 displays the display image data that isreceived from the image display memory 24 via the D/A converter 26. Inthis case, the image display unit 28 is in a live view display state. Inother words, the image display unit 28 performs an electronic viewfinderoperation.

In step S1402, the imaging apparatus determines whether moving imagerecording start is selected by the operation unit 70. If it isdetermined that the moving image recording start is not selected (NO instep S1402), then the processing returns to step S1401 and the imagingapparatus repeats the above-described processing. If it is determinedthat the moving image recording start is selected (YES in step S1402),then in step S1403, the imaging apparatus starts the moving imagerecording.

When the moving image recording starts, the audio signal processingcircuit 33 encodes audio data input from the microphone (notillustrated) and the encoded audio data is temporarily stored in anaudio encoded data buffer of the memory 30.

The amount of temporarily stored moving image encoded data and audioencoded data increases with elapsing time. Therefore, the temporarilystored encoded data are converted into a predetermined file format andoccasionally written into the recording medium 200 via the interface 90.

In step S1404, the imaging apparatus determines whether moving imagerecording stop is selected by the operation unit 70. If it is determinedthat the moving image recording stop is not selected (NO in step S1404),then the processing returns to step S1403 and the imaging apparatuscontinues the moving image recording. If it is determined that themoving image recording stop is selected (YES in step S1404), then instep S1405, the imaging apparatus determines whether live view stop isselected.

If it is determined that the live view stop is not selected (NO in stepS1405), the processing returns to step S1402 and the imaging apparatuswaits for the next moving image recording. If it is determined that thelive view stop is selected (YES in step S1405), the imaging apparatusterminates the moving image capturing processing routine.

Next, an example of moving image file generation is described below. Ifa moving image recording button is turned on in a moving image capturingmode, the imaging apparatus starts moving image capturing processing.First, the imaging apparatus generates a new file that includes moov(i.e., meta data BOX) and mdat (i.e., media data BOX) of the initialfragment.

Next, the imaging apparatus generates dust position correction data. Thedust position correction data stores lens information of the lens usedin the moving image capturing processing, such as aperture value, lenspupil position information, and the dust correction data illustrated inFIG. 5.

The memory 52 stores the generated dust position correction data. Theimaging apparatus reads the dust position correction data from thememory 52 and writes the read data in the meta data moov of the presentfragment.

FIG. 14 is a flowchart illustrating an example of recording processingto be performed for each frame in the moving image capturing operationaccording to the present exemplary embodiment. The system controlcircuit 50 executes a moving image capturing processing program, whichcan be loaded from the memory 52, to realize the above-describedprocessing.

First, in step S1501, the imaging apparatus determines whether anelectronic image stabilization function was turned on in the movingimage capturing operation. If it is determined that the electronic imagestabilization function was turned on (YES in step S1501), then in stepS1502, the imaging apparatus detects a shake amount based on an outputof the angular speed sensor (e.g., a vibration gyroscope).

If it is determined that the electronic image stabilization function wasturned off in the moving image capturing operation (NO in step S1501),the processing proceeds to step S1505. As the image clipping position isfixed, the imaging apparatus does not perform recording of the clippingposition in a moving image data file.

In step 1503, the imaging apparatus calculates a clipping positioncorrection amount to cancel the shake amount, and corrects a clippingposition when an image signal is read by the image sensor 14.

In step S1504, the imaging apparatus records corrected image clippingposition information in the moving image data file. The clippingposition information includes an X coordinate value and a Y coordinatevalue, which are determined relative to the origin, representing theclipping position on the entire screen of the image sensor 14, asillustrated in FIG. 16.

Alternatively, the clipping position information may include a deviation(i.e., a difference amount) relative to a reference frame as illustratedin FIG. 17. More specifically, in FIG. 17, the position of the referenceframe is determined relative to the origin, and the position of thepresent frame is determined relative to the position of the referenceframe. To perform file recording, the imaging apparatus records theimage clipping position information in the meta data moof of the presentfragment.

As image data of a plurality of frames can be recorded in one fragment,the imaging apparatus adds a plurality pieces of clipping positioninformation in the meta data moof of one fragment. Therefore, in thewriting operation, it is required to associate the frame with theclipping position information.

In step S1505, the imaging apparatus performs image encoding processingto reduce a data amount of the image data. In step S1506, the imagingapparatus records the compressed image data into a file.

Sequential processing in steps S1501 to S1506 is for recording one-frameimage. Therefore, the imaging apparatus repetitively performs theabove-described sequential processing during the moving image recordingoperation.

In FIG. 14, the imaging apparatus determines ON/OFF of the electronicimage stabilization function for each frame (see step S1501).Alternatively, the imaging apparatus may perform the above-describeddetermination only when it starts the moving image capturing operationand may use the obtained result in the branch processing.

An example of image processing for removing an influence of dust frommoving image data captured through an electronic image stabilizationsystem is described below.

The imaging apparatus executes the dust removal processing for eachframe using the dust correction data (i.e., dust information) recordedin the recording medium 200 in association with the image data, althoughthe rest of the processing is similar to that for a still image.

The dust correction data is stored in the moov of a moving image file.However, as the number of the dust information is only one, dustinformation is converted for each frame.

FIG. 15 is a flowchart illustrating dust removal processing to beperformed for each frame according to the present exemplary embodiment.

The camera or a separately provided image processing apparatus mayperform the dust removal processing.

In the present exemplary embodiment, in step S1601, the imagingapparatus determines whether a frame of an image to be subjected to thedust removal processing was captured in an electronic imagestabilization ON state.

If it is determined that the image was captured in the electronic imagestabilization ON state (YES in step S1601), then instep S1602, theimaging apparatus acquires the clipping position information of theframe recorded in the meta data moof.

If it is determined that the image was captured in an electronic imagestabilization OFF state (NO instep S1601), the imaging apparatusdirectly performs dust correction processing.

In step S1603, as illustrated in FIG. 18, the imaging apparatus convertsdust position coordinates to cancel a variation in the clipping positionbased on the clipping position information acquired in step S1602.

In step S1604, the imaging apparatus performs the dust correctionprocessing using the converted dust position coordinates. The dustcorrection processing to be performed in step S1604 is the sequentialprocessing illustrated in FIG. 8.

As described above, in the present exemplary embodiment, the imagingapparatus records the image clipping position into a moving image filefor each frame when the moving image capturing operation is performed inthe electronic image stabilization ON state. Furthermore, when theimaging apparatus performs the dust removal processing, the imagingapparatus converts dust position information based on the clippingposition recorded for each frame before performing the dust removalprocessing.

In this manner, the imaging apparatus can perform the dust removalprocessing on a moving image when the electronic image stabilizationsystem suppresses a camera shake component of the moving image in themoving image recording operation.

A digital camera according to a second exemplary embodiment is similarto that according to the first exemplary embodiment. Differences inoperation between the first exemplary embodiment and the secondexemplary embodiment are described below.

The second exemplary embodiment is different from the first exemplaryembodiment in that the imaging apparatus performs recording only when avariation in the clipping position exceeds a predetermined threshold(hereinafter, referred to as TH), when the image clipping positioninformation is recorded in a file.

FIG. 19 is a flowchart illustrating an example of recording processingaccording to the present exemplary embodiment, which is executed foreach frame in the moving image capturing operation.

As described in the interpolation routine illustrated in FIG. 8, an areaincluding the circle center coordinate Di can be set to have anappropriate width in the dust area determination. Therefore, if theshift amount of the image clipping position is small, the imagingapparatus can perform the dust removal processing without recording theshift amount.

Therefore, if it is determined that the shift amount in the imageclipping position is greater than the threshold TH (YES in step S1605),then in step S1504, the imaging apparatus records the image clippingposition information in the moving image file.

If it is determined that the variation in the image clipping position isequal to or less than the threshold TH (equal to or less than apredetermined value) (NO in step S1605), the imaging apparatus does notrecord the image clipping position information in the moving image file.

However, in a case where the clipping position information is notrecorded for all frames, each frame is associated with the clippingposition information. Therefore, when the imaging apparatus performsrecording of the clipping position information in step S1504,information is also recorded indicating a corresponding frame (e.g.,position of the track in the order from the head).

As described above, the present exemplary embodiment can reduce the dataamount when the camera shake amount is small and can reduce theprocessing amount in the conversion of dust position coordinates in thedust removal processing.

The following method can be used to implement the exemplary embodiments.A storage medium storing a software program code for realizing thefunctions of the above-described exemplary embodiments can be suppliedto a system or an apparatus. The program code includes computerexecutable instructions for implementing the embodiments of the presentinvention. A computer (or CPU or micro-processing unit (MPU)) in thesystem or the apparatus can execute the program code stored in thestorage medium.

In this case, the program code itself read out of the storage medium andexecuted by the computer realizes the functions of the above-describedexemplary embodiments. The storage medium storing the program codeconstitutes the present invention. Not only the functions of theabove-described exemplary embodiments can be realized by the computerthat executes the read program code, but also the present invention canbe realized, for example, in the following cases.

An operating system (OS) or other application software running on acomputer can execute part or all of actual processing based oninstructions of the program code to realize the functions of theabove-described exemplary embodiments.

Additionally, the program code read out of a storage medium can bewritten into a memory of a function expansion card inserted in acomputer or into a memory of a function expansion unit connected to thecomputer. In this case, based on instructions of the program code, a CPUprovided on the function expansion card or the function expansion unitcan execute part or all of the processing to realize the functions ofthe above-described exemplary embodiments.

When the present invention is applied to the above-described storagemedium, a program code corresponding to the above-described procedure isstored in the storage medium.

A wide variety of storage media may be used to store the program code.The storage medium may be, for example, any of a flexible disk (floppydisk), a hard disk, an optical disk, a magneto-optical disk, a compactdisc (CD), a digital versatile disc (DVD), a read only memory (ROM), aCD-recordable (R), a CD-rewritable, a DVD-recordable, a DVD-rewritable,a magnetic tape, a nonvolatile memory card, a flash memory device, andso forth.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all modifications, equivalent structures and functions.

1. An imaging apparatus, comprising: an image capturing unit includingan image sensor capable of photoelectrically converting a subject imageand configured to generate moving image data based on an output signalof the image sensor; a first acquisition unit configured to acquireforeign substance information including information relating to at leasta position and a size of a foreign substance adhered to an opticalelement disposed on a front side of the image sensor; an image clippingunit configured to clip a predetermined area on an entire screen of theimage sensor and to output image clipping information indicating aposition of the predetermined area; and a recording unit configured torecord the foreign substance information and the image clippinginformation in association with the moving image data.
 2. The imagingapparatus according to claim 1, wherein the image clipping informationincludes an X coordinate and a Y coordinate on the entire screen of theimage sensor.
 3. The imaging apparatus according to claim 1, wherein theimage clipping information is a different relative to an image clippingposition of a reference frame.
 4. The imaging apparatus according toclaim 3, wherein the recording unit does not perform recording of theimage clipping information in a case where the difference in the imageclipping position is equal to or less than predetermined value.
 5. Theimaging apparatus according to claim 1, further comprising: a secondacquisition unit configured to acquire the foreign substance informationand the image clipping information which are recorded in associationwith each other; and an interpolation unit configured to convert theforeign substance information based on the image clipping informationand perform interpolation processing on a pixel corresponding to theforeign substance of the moving image data using the converted foreignsubstance information.
 6. An imaging apparatus, comprising: an imagecapturing unit including an image sensor capable of photoelectricallyconverting a subject image and configured to generate moving image databased on an output signal of the image sensor; a first acquisition unitconfigured to acquire foreign substance information includinginformation relating to at least a position and a size of a foreignsubstance adhered to an optical element disposed on a front side of theimage sensor; an image clipping unit configured to clip a predeterminedarea on an entire screen of the image sensor and to output imageclipping information indicating a position of the predetermined area;and an interpolation unit configured to convert the foreign substanceinformation based on the image clipping information and performinterpolation processing on a pixel corresponding to the foreignsubstance of the moving image data using the converted foreign substanceinformation.
 7. A method for controlling an imaging apparatus includingan image capturing unit having an image sensor capable ofphotoelectrically converting a subject image and configured to generatemoving image data based on an output signal of the image sensor, themethod comprising: storing foreign substance information includinginformation relating to at least a position and a size of a foreignsubstance adhered to an optical element disposed on a front side of theimage sensor; clipping a predetermined of area on an entire screen ofthe image sensor and to output image clipping information indicating aposition of the predetermined area; and recording the foreign substanceinformation and the image clipping information in association with themoving image data.
 8. A method for controlling an imaging apparatusincluding an image capturing unit having an image sensor capable ofphotoelectrically converting a subject image and configured to generatemoving image data based on an output signal of the image sensor, themethod comprising: storing foreign substance information includinginformation relating to at least a position and a size of a foreignsubstance adhered to an optical element disposed on a front side of theimage sensor; clipping a predetermined area on an entire screen of theimage sensor and to output image clipping information indicating aposition of the predetermined area; and converting the foreign substanceinformation based on the image clipping information and performinterpolation processing on a pixel corresponding to the foreignsubstance of the moving image data using the converted foreign substanceinformation.
 9. A program recorded on a non-transitory computer-readablestorage medium containing computer-executable instructions for causing acomputer to execute a method for controlling an imaging apparatus thatincludes an image capturing unit having an image sensor capable ofphotoelectrically converting a subject image and configured to generatemoving image data based on an output signal of the image sensor, themethod comprising: storing foreign substance information includinginformation relating to at least a position and a size of a foreignsubstance adhered to an optical element disposed on a front side of theimage sensor; clipping a predetermined area on an entire screen of theimage sensor and to output image clipping information indicating aposition of the predetermined area; and recording the foreign substanceinformation and the image clipping information in association with themoving image data.
 10. A program recorded on a non-transitorycomputer-readable storage medium containing computer-executableinstructions for causing a computer to execute a method for controllingan imaging apparatus that includes an image capturing unit having animage sensor capable of photoelectrically converting a subject image andconfigured to generate moving image data based on an output signal ofthe image sensor, the method comprising: storing foreign substanceinformation including information relating to at least a position and asize of a foreign substance adhered to an optical element disposed on afront side of the image sensor; clipping a predetermined area on anentire screen of the image sensor and to output image clippinginformation indicating a position of the predetermined area; andconverting the foreign substance information based on the image clippinginformation and perform interpolation processing on a pixelcorresponding to the foreign substance of the moving image data usingthe converted foreign substance information.